Silverback: Scalable Association Mining For Massive Temporal Data in Columnar Probabilistic Databases

We investigate large scale probabilistic association mining on modest hardware infrastructure. We first propose a probabilistic columnar infrastructure for storing the transaction database. Using Bloom filters and reservoir sampling techniques, the storage is e cient and probabilistic. Then we propose an accurate probabilistic algorithm for mining frequent item-sets. Our algorithm relies on the Apriori principle but has a novel probabilistic pruning technique, which reduces frequent item-set candidates without counting every candidate’s support size. In the experiments, our Silverback framework, with satisfying accuracy, outperforms Hadoop Apriori implementation in terms of run time. Silverback has been commercially deployed and developed at Voxsup Inc. since May 2011. Northwestern Tech Report July 13, 2013